Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

In the paper we presented the guidelines for the application of nonparametric statistical tests and post hoc procedures devised to perform multiple comparisons of machine learning algorithms. We emphasized it is necessary to distinguish between pairwise and multiple comparison tests. We showed that the pairwise Wilcoxon test, when employed to multiple comparisons, would lead to overoptimistic conclusions. We carried out intensive normality examination employing different ten tests showing that the output of machine learning algorithms for regression problems do not satisfy normality requirements. We conducted experiments on nonparametric statistical tests and post hoc procedures designed for multiple 1×N and N×N comparisons with six different neural regression algorithms over 29 benchmark regression data sets. Our investigation proved the usefulness and strength of multiple comparison statistical procedures to analyse and select machine learning algorithms.